Visual Studio 2010 Testing Overview - based on 2010 Beta 2

1. Testing with Visual Studio 2010 Breaking the walls between developers and testers Steve Lange Developer Technology Specialist Microsoft | Denver, CO stevenl@microsoft.com | blogs.msdn.com/slange

2. Agenda Introductions Microsoft’s Approach to Testing Test Capabilities in Visual Studio 2010 Test Case Management Test Plans, Test Suites, & Test Cases Lab Management Rich & Actionable Bugs Perform Exploratory testing Coded UI Test 2

3. The State Of SW Development An Acceptable Rate of Failure? Based on time, budget, and delivering specified functionality 68% of projects never made it into production or were cancelled 1 Can we continue to support this level of Success?2 Extreme Chaos, The Standish Group International, Inc. – 2009 Average Costoverrun: 45% Average Timeoverrun: 63% Functionalitydelivered on average: 67% 2. Extreme Chaos, The Standish Group International, Inc - 2009 Source: 1. Dave West, Forrester Research

4. Frustrations in development teams 4 “my testers are spending too long testing the same thing” “tooling is expensive (time, licenses, people)” “the developers say the defects are useless” “when is my software ready to ship?” “we have a changed requirement, what do I need to test?” “developers and testers work in silos and don’t communicate/speak the same language”

5. Challenges QA is not aligned with lifecycle Developers and testers work in silos QA is not leveraged

6. Microsoft Approach

7. Tester Segmentation 70% of testing happens here majority of test tools target here black box testing white box testing api testing

12. Goals for 2010 Release - Testing Align QA with the Lifecycle Create tighter Dev/Test interaction Enable highly leveraged QA teams

13. VS 2010 Test Capabilities generalist specialist coded UI test unit testing web performance test test runner load test test case management virtual lab management data diagnostic adapters (video, action log, event log etc) team foundation server with reporting (bugs, requirements, user stories, source control, build)

14. Visual Studio 2010 Test Components Server Components Client Components Lab Management SCVMM VM Host Library Shares Agent Team Foundation Server (TFS) Visual Studio 2010 Load Agent Lab Management Microsoft Test & Lab Manager (MTLM) Visual Studio Pro TFS Web Access Test Case Management Controller Build Management Load Agent Agent Work Item Tracking Source Control Agent Collector Agent Target Server Build Server Build Server Updated New

15. Visual Studio 2010

16. Key Test Capabilities in VS 2010 16

17. Key Test Capabilities in VS2010

18. Test Case Management 18

19. Test Case Management Create, organize, and run test cases Manage a suite hierarchy Includes “Dynamic” suites Configure active iterations First-class Work Items in TFS Can be data-driven 19

20. Microsoft Confidential 20 Test Case is first class work item in TFS Detailed steps along with expected results

21. Microsoft Confidential 21 Data drive a test case Specify data in parameter table

22. Microsoft Confidential 22 Select common steps and convert to Shared Steps

23. Microsoft Confidential 23 Shared Steps allows for common steps to be reused across test cases

24. Walkthrough Test Case Management

25. Microsoft Confidential 25

26. A tester working in Test and Lab manager can see requirements and tests side-by-side Microsoft Confidential 26

27. He can add existing tests cases to a requirement… Microsoft Confidential 27

28. …and create new ones with test steps! Microsoft Confidential 28

29. He runs the test.. Microsoft Confidential 29

30. Which starts up the Test Runner… Microsoft Confidential 30

31. He can go through the test case step-by-step Microsoft Confidential 31

32. ..and create rich, actionable bugs can be created right from the test… Microsoft Confidential 32

33. …automatically capturing all the system information… Microsoft Confidential 33

34. …and the exact steps performed! Microsoft Confidential 34

35. Back, in Visual Studio, the development team can see the rich bug… Microsoft Confidential 35

36. …the linked test case... Microsoft Confidential 36

37. …and the requirement for full traceability! Microsoft Confidential 37

38. All the while, the team can see progress on requirements as well as the quality of the requirements!

39. Lab Management 39

40. Lab Management Simplifies environment setup Improving deployment quality Simplifies testing multiple configurations Enables reverting to baseline configuration Improves test hardware utilization

41. Lab Management Architecture System Center Virtual Machine Manager Team Foundation Server (TFS) Lab Management Test Case Management Microsoft Test & Lab Manager (MTLM) Build Management Hyper-V Hosts VM Library Shares Work Item Tracking VM 1 Share 1 Source Control VM 2 Share 2 VM 3 Share 3 Updated New VM 4 Share 4 41

42. Microsoft Confidential 42 Create Virtual Environment from Lab Center

43. Microsoft Confidential 43 Specify name and location for environment

44. Microsoft Confidential 44 Pick role for VM Select VM from template library

45. Microsoft Confidential 45 Specify capabilities – “Run Tests” ; “Run Workflows” ; “Network Isolation”

46. Microsoft Confidential 46 Environment (collection of VMs) created

47. Microsoft Confidential 47 Manage snapshots for environment Access environment using Lab Environment Viewer

48. Lab Management for Distributed Team Lower cost of ownership Now have a visual face to the lab! No waiting - Create machines on a remote site without assistance Get traction of idle machines using “last accessed time” Friction free – Mark and reserve your machines

49. View your lab

50. View your lab – Who owns what?

51. View your lab – Where are the machines?

52. View your lab – who has idle machines?

53. Rich & Actionable Bugs 53

54. Rich & Actionable Bugs 54

55. Walkthrough Filing Rich & Actionable Bugs

56. Microsoft Confidential 56 Dev clicks on lvr link & selects connect to snapshot of the environment

57. Microsoft Confidential 57 Dev is able to access the snapshot’ed environment via LE viewer

58. Exploratory Testing 58

59. Exploratory Testing “Wander” application looking for issues Errors, UX, etc. Turn problems into bugs, then test cases Simple, 1-step test case 59

60. Walkthrough Exploratory Testing

61. Coded UI Test Microsoft Confidential 61

62. Coded UI Test Puts recorded actions into code Dissects action against executing code Can be associated against test case Microsoft Confidential 62

63. Microsoft Confidential 63 From Test->new test, click Coded UI test, specify test name and project name

64. Microsoft Confidential 64 Use existing action recording to generate code

65. Microsoft Confidential 65 Specify test case which has action recording and click OK

66. Microsoft Confidential 66 Code generated from action recording – datasource as TestCase

67. Microsoft Confidential 67 Code generated handles data parameterization

68. Microsoft Confidential 68 Invoke Coded UI Test Builder for adding assertions

69. Microsoft Confidential 69 Drag cross-hair onto the control you want to add assertions on

70. Microsoft Confidential 70 Properties of selected control and its UIMap is shown

71. Microsoft Confidential 71 Select property to assert on and right click to add assertion

72. Microsoft Confidential 72 Add the assert

73. Microsoft Confidential 73 Specify Assert method name and generate code

74. Microsoft Confidential 74 Code generated for assert method

75. Microsoft Confidential 75 Add solution to source control

76. Microsoft Confidential 76 Check-in pending changes

77. Microsoft Confidential 77 Go to test case that you want to associate the newly created automation

78. Microsoft Confidential 78 Specify test method and click OK; save test case

79. Reporting Microsoft Confidential 79

80. Reporting binds the team Query Work Items Excel reports Dashboards

81. Create reports in one click

82. One Click reports – Team Test

83. Show me the test cases that found “good” bugs

84. QA Dashboard

85. Rich Bugs

86. Summary 86

87. Summary 87

88. Thank You 88

89. Contact Information 89

90. Contact Information 90